Skip to content

q4k mmq optimizations - #80

Merged
jimw567 merged 8 commits into
gfx11from
lichang.q4k-opt
Aug 25, 2026
Merged

q4k mmq optimizations#80
jimw567 merged 8 commits into
gfx11from
lichang.q4k-opt

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Overview

MUL_MAT shape (m × n × k) Baseline lichang.q4k-opt Improvement q4k-opt throughput
1024 × 128 × 4096 85.54 µs 65.42 µs +23.52% 16.41 TFLOPS
5376 × 128 × 2048 152.08 µs 142.75 µs +6.13% 19.74 TFLOPS
4096 × 128 × 4096 216.61 µs 201.98 µs +6.75% 21.26 TFLOPS
8192 × 128 × 4096 360.98 µs 346.59 µs +3.99% 24.78 TFLOPS
12288 × 128 × 4096 527.31 µs 504.85 µs +4.26% 25.52 TFLOPS
4096 × 512 × 14336 2549.82 µs 2490.45 µs +2.33% 24.14 TFLOPS
Test Group TTFT improvement Decode improvement Baseline Candidate >5% regression
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128 standard_128 1.88% -0.03% PASS PASS
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 0.26% -0.26% PASS PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128 standard_128 5.38% 0.01% PASS PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 2.06% -0.13% PASS PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128 standard_128 1.61% 0.46% PASS PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968 standard_long_context 0.79% -0.15% PASS PASS
Qwen3-1.7B_Q4_K_M_GGUF_128 standard_128 1.67% -0.03% PASS PASS
Qwen3-1.7B_Q4_K_M_GGUF_3968 standard_long_context 1.09% -0.10% PASS PASS
Qwen3-4B_Q4_K_M_GGUF_128 standard_128 2.39% not rerun PASS PASS
Qwen3-4B_Q4_K_M_GGUF_3968 standard_long_context 1.56% 0.27% PASS PASS
Qwen3-8B_Q4_K_M_GGUF_128 standard_128 1.87% 0.09% PASS PASS
Qwen3-8B_Q4_K_M_GGUF_3968 standard_long_context 0.14% 0.18% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_128 standard_128 1.57% 0.03% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_128 standard_128 3.42% 0.06% PASS PASS
Llama-2-7B_Q4_K_M_GGUF_128 standard_128 1.56% 3.36% PASS PASS
Llama-2-7B_Q4_K_M_GGUF_1920 standard_long_context 0.37% 0.03% PASS PASS
Gemma-2B_Q4_K_M_GGUF_128 standard_128 2.32% -0.08% PASS PASS
Gemma-2B_Q4_K_M_GGUF_8000 standard_long_context -0.59% 0.01% PASS PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128 standard_128 1.18% -0.05% PASS PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000 standard_long_context 0.32% -0.07% PASS PASS
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128 standard_128 -0.77% 0.19% PASS PASS
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128 standard_128 0.49% not rerun PASS PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_128 standard_128 2.76% -0.07% PASS PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968 standard_long_context 0.31% 0.06% PASS PASS
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128 standard_128 3.67% not rerun PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_128 standard_128 0.24% not rerun PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_4096 standard_long_context 0.16% -0.02% PASS PASS
Gemma-3-4B-IT_VLM standard_other 0.38% -0.07% PASS PASS
Qwen2.5-VL-3B-Instruct_VLM standard_other 0.30% -0.00% PASS PASS
Qwen2.5-VL-7B-Instruct_VLM standard_other 0.04% -0.01% PASS PASS
Qwen3-VL-4B-Instruct_VLM standard_other 0.41% 0.09% PASS PASS
Gemma-4-26B-A4B-IT_VLM standard_other 0.14% -0.01% PASS PASS
Gemma-4-31B-IT_VLM standard_other 0.68% 0.06% PASS PASS
MiniCPM-V-2_6_VLM standard_other 0.65% 0.06% PASS PASS
MiniCPM-o-2_6_VLM standard_other 0.75% 0.15% PASS PASS
Janus-Pro-1B_VLM standard_other 2.42% -0.00% PASS PASS
Janus-Pro-7B_VLM standard_other 1.43% -0.05% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_4096 standard_long_context 0.61% -0.03% PASS PASS
GLM-4.7-Flash_Q4_K_M_GGUF_128 standard_128 -0.15% 0.11% PASS PASS
Gemma-4-12B-it_Q4_K_M_GGUF_128 standard_128 -0.04% -0.03% PASS PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM standard_other 0.81% 0.10% PASS PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_4096 standard_long_context 0.85% 0.10% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_prose mtp_baseline -0.93% -0.27% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_prose mtp_verify -0.15% 0.18% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_code mtp_baseline -1.49% -0.29% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_code mtp_verify -1.34% 0.30% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_list mtp_baseline -1.37% -0.02% PASS PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_list mtp_verify -0.89% 0.25% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_prose mtp_baseline -0.95% -0.06% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_prose mtp_verify -1.52% -0.07% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_code mtp_baseline -1.81% -0.06% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_code mtp_verify -1.70% -0.16% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_list mtp_baseline -1.73% -0.17% PASS PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_list mtp_verify 0.45% -0.43% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_prose mtp_baseline -0.25% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_prose mtp_verify 0.01% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_code mtp_baseline 0.86% 0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_code mtp_verify 0.17% -0.03% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_list mtp_baseline -0.32% -0.01% PASS PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_list mtp_verify -0.68% 0.30% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_prose mtp_baseline 0.93% 0.04% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_prose mtp_verify 1.09% 0.09% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_code mtp_baseline 1.19% 0.04% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_code mtp_verify 0.38% -0.07% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_list mtp_baseline 0.53% 0.05% PASS PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_list mtp_verify -0.65% -0.42% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_prose mtp_baseline 0.29% 0.01% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_prose mtp_verify -0.17% 0.04% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_code mtp_baseline -0.49% -0.01% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_code mtp_verify 0.94% -0.02% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_list mtp_baseline 0.21% 0.06% PASS PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_list mtp_verify -0.00% -0.11% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_prose mtp_baseline 0.79% 0.22% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_prose mtp_verify 0.96% -0.37% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_code mtp_baseline 0.09% -0.09% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_code mtp_verify 0.13% -0.43% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_list mtp_baseline 1.31% -0.17% PASS PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_list mtp_verify -0.97% -0.13% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_prose mtp_baseline 3.48% 0.03% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_prose mtp_verify 0.35% -0.12% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_code mtp_baseline 1.37% 0.06% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_code mtp_verify -0.30% -0.06% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_list mtp_baseline 1.30% -0.01% PASS PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_list mtp_verify 0.31% -0.66% PASS PASS

Additional information

Requirements

liangliangchang and others added 5 commits July 31, 2026 13:18
@liangliangchang @cursoragent
HIP: optimize RDNA3.5 Q4_K MMQ pipeline
2815b3e 
Pipeline Q4_K tile loads with split WMMA dequantization and select the measured tile width by matrix shape.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: fix RDNA3.5 Q4_K pipeline integration
a06b975 
Match the refactored 32-row warp layout and limit the specialized J64 path to the small shape where it provides a stable gain.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: tune Q4_K wave mapping by tile width
c83bdac 
Use the generic vec-dot with J64 two-row waves and J128 one-row waves, retaining the J64 prefetch loop for the best measured shape coverage.

Assisted-by: GPT-5.6 Sol
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: preserve expert-aware Q4_K MMQ tiling
bec4a25 
Avoid overriding MoE's per-expert tile selection with the dense Q4_K pipeline policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: remove Q4_K MMQ test override
f1cd056 
Keep production dispatch shape-driven and avoid exposing an experimental tile-width environment knob.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang
liangliangchang marked this pull request as ready for review August 12, 2026 17:56
liangliangchang and others added 2 commits August 12, 2026 13:11
@liangliangchang @cursoragent
HIP: extend RDNA3.5 Q4_K prefetch to J128
f602551 
Reuse the validated next-weight pipeline for J128 to improve large Q4_K MMQ throughput without retaining experimental pacing controls.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: batch RDNA3.5 Q4_K J128 WMMA phases
96fc67b 
Separate low and high nibble WMMA phases across J128 tiles to retain the faster dequantization schedule with next-weight prefetch.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mgehre-amd

mgehre-amd commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Is the regression on Qwen3.5-4B_Q4_K_M_GGUF_baseline_code reproducible? If yes, any idea how to avoid it?

mgehre-amd
mgehre-amd reviewed Aug 17, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh
mgehre-amd
mgehre-amd reviewed Aug 17, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh
mgehre-amd
mgehre-amd reviewed Aug 17, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh Outdated
@mgehre-amd

mgehre-amd commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Is this covered by unit tests in tests/test-backend-ops.cpp?

mgehre-amd
mgehre-amd approved these changes Aug 17, 2026
View reviewed changes

@mgehre-amd mgehre-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick read and besides some comments, it looks good to me.
I'll defer to @jimw567 or @Annieren for a more in-depth review.

jimw567
jimw567 approved these changes Aug 24, 2026
View reviewed changes

@jimw567 jimw567 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@liangliangchang @cursoragent
HIP: gate RDNA3.5 Q4_K WMMA on RDNA3_5 and test n>128
c73fa02 
Compile the Q4_K batched WMMA path under RDNA3_5 only, and add a 4096x1024x12288 Q4_K MMQ case so occupancy-selected J=128 is covered.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang

liangliangchang commented Aug 24, 2026

Copy link
Copy Markdown
Author

Is the regression on Qwen3.5-4B_Q4_K_M_GGUF_baseline_code reproducible? If yes, any idea how to avoid it?

It is not reproducible. Also, baseline code uses input token size 32, it won't go through the path added in this PR.

@liangliangchang

liangliangchang commented Aug 24, 2026

Copy link
Copy Markdown
Author

Is this covered by unit tests in tests/test-backend-ops.cpp?

It is covered by cases already in. Just added one more with larger N.

@jimw567
jimw567 merged commit 03d2068 into gfx11 Aug 25, 2026
8 checks passed
@jimw567
jimw567 deleted the lichang.q4k-opt branch August 25, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jimw567 jimw567 jimw567 approved these changes
@mgehre-amd mgehre-amd mgehre-amd approved these changes
@Annieren Annieren Awaiting requested review from Annieren
@roberteg16 roberteg16 Awaiting requested review from roberteg16

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liangliangchang @mgehre-amd @jimw567